{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monitoring optimisation\n",
    "\n",
    "In this notebook we'll demo how to use `gpflow.training.monitor` for logging the optimisation of a GPflow model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "import os\n",
    "import numpy as np\n",
    "import gpflow\n",
    "import gpflow.training.monitor as mon\n",
    "import numbers\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the GPflow model\n",
    "We first generate some random data and create a GPflow model.\n",
    "\n",
    "Under the hood, GPflow gives a unique name to each model which is used to name the Variables it creates in the TensorFlow graph containing a random identifier. This is useful in interactive sessions, where people may create a few models, to prevent variables with the same name conflicting. However, when loading the model, we need to make sure that the names of all the variables are exactly the same as in the checkpoint. This is why we pass name=\"SVGP\" to the model constructor, and why we use gpflow.defer_build()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "X = np.random.rand(10000, 1) * 10\n",
    "Y = np.sin(X) + np.random.randn(*X.shape)\n",
    "Xt = np.random.rand(10000, 1) * 10\n",
    "Yt = np.sin(Xt) + np.random.randn(*Xt.shape)\n",
    "\n",
    "with gpflow.defer_build():\n",
    "    m = gpflow.models.SVGP(X, Y, gpflow.kernels.RBF(1), gpflow.likelihoods.Gaussian(),\n",
    "                           Z=np.linspace(0, 10, 5)[:, None],\n",
    "                           minibatch_size=100, name=\"SVGP\")\n",
    "    m.likelihood.variance = 0.01\n",
    "m.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compute log likelihood before the optimisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LML before the optimisation: -1271605.621944\n"
     ]
    }
   ],
   "source": [
    "print('LML before the optimisation: %f' % m.compute_log_likelihood())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using a TensorFlow optimiser. All TensorFlow optimisers have a support for `global_step` variable. Its purpose is to track how many optimisation steps have occurred. It is useful to keep this in a TensorFlow variable as this allows it to be restored together with all the parameters of the model.\n",
    "\n",
    "The code below creates this variable using a monitor's helper function. It is important to create it before building the monitor in case the monitor includes a checkpoint task. This is because the checkpoint internally uses the TensorFlow Saver which creates a list of variables to save. Therefore all variables expected to be saved by the checkpoint task should exist by the time the task is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = m.enquire_session()\n",
    "global_step = mon.create_global_step(session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construct the monitor\n",
    "\n",
    "Next we need to construct the monitor. `gpflow.training.monitor` provides classes that are building blocks for the monitor. Essengially, a monitor is a function that is provided as a callback to an optimiser. It consists of a number of tasks that may be executed at each step, subject to their running condition.\n",
    "\n",
    "In this example, we want to:\n",
    "- log certain scalar parameters in TensorBoard,\n",
    "- log the full optimisation objective (log marginal likelihood bound) periodically, even though we optimise with minibatches,\n",
    "- store a backup of the optimisation process periodically,\n",
    "- log performance for a test set periodically.\n",
    "\n",
    "We will define these tasks as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_task = mon.PrintTimingsTask().with_name('print')\\\n",
    "    .with_condition(mon.PeriodicIterationCondition(10))\\\n",
    "    .with_exit_condition(True)\n",
    "\n",
    "sleep_task = mon.SleepTask(0.01).with_name('sleep').with_name('sleep')\n",
    "\n",
    "saver_task = mon.CheckpointTask('./monitor-saves').with_name('saver')\\\n",
    "    .with_condition(mon.PeriodicIterationCondition(10))\\\n",
    "    .with_exit_condition(True)\n",
    "\n",
    "file_writer = mon.LogdirWriter('./model-tensorboard')\n",
    "\n",
    "model_tboard_task = mon.ModelToTensorBoardTask(file_writer, m).with_name('model_tboard')\\\n",
    "    .with_condition(mon.PeriodicIterationCondition(10))\\\n",
    "    .with_exit_condition(True)\n",
    "\n",
    "lml_tboard_task = mon.LmlToTensorBoardTask(file_writer, m).with_name('lml_tboard')\\\n",
    "    .with_condition(mon.PeriodicIterationCondition(100))\\\n",
    "    .with_exit_condition(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the above code shows, each task can be assigned a name and running conditions. The name will be shown in the task timing summary.\n",
    "\n",
    "There are two different types of running conditions: `with_condition` controls execution of the task at each iteration in the optimisation loop. `with_exit_condition` is a simple boolean flag indicating that the task should also run at the end of optimisation.\n",
    "In this example we want to run our tasks periodically, at every iteration or every 10th or 100th iteration.\n",
    "\n",
    "Notice that the two TensorBoard tasks will write events into the same file. It is possible to share a file writer between multiple tasks. However it is not possible to share the same event location between multiple file writers. An attempt to open two writers with the same location will result in error.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom tasks\n",
    "We may also want to perfom certain tasks that do not have pre-defined `Task` classes. For example, we may want to compute the performance on a test set. Here we create such a class by extending `BaseTensorBoardTask` to log the testing benchmarks in addition to all the scalar parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTensorBoardTask(mon.BaseTensorBoardTask):\n",
    "    def __init__(self, file_writer, model, Xt, Yt):\n",
    "        super().__init__(file_writer, model)\n",
    "        self.Xt = Xt\n",
    "        self.Yt = Yt\n",
    "        self._full_test_err = tf.placeholder(gpflow.settings.float_type, shape=())\n",
    "        self._full_test_nlpp = tf.placeholder(gpflow.settings.float_type, shape=())\n",
    "        self._summary = tf.summary.merge([tf.summary.scalar(\"test_rmse\", self._full_test_err),\n",
    "                                         tf.summary.scalar(\"test_nlpp\", self._full_test_nlpp)])\n",
    "    \n",
    "    def run(self, context: mon.MonitorContext, *args, **kwargs) -> None:\n",
    "        minibatch_size = 100\n",
    "        preds = np.vstack([self.model.predict_y(Xt[mb * minibatch_size:(mb + 1) * minibatch_size, :])[0]\n",
    "                            for mb in range(-(-len(Xt) // minibatch_size))])\n",
    "        test_err = np.mean((Yt - preds) ** 2.0)**0.5\n",
    "        self._eval_summary(context, {self._full_test_err: test_err, self._full_test_nlpp: 0.0})\n",
    "\n",
    "        \n",
    "custom_tboard_task = CustomTensorBoardTask(file_writer, m, Xt, Yt).with_name('custom_tboard')\\\n",
    "    .with_condition(mon.PeriodicIterationCondition(100))\\\n",
    "    .with_exit_condition(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can put all these tasks into a monitor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "monitor_tasks = [print_task, model_tboard_task, lml_tboard_task, custom_tboard_task, saver_task, sleep_task]\n",
    "monitor = mon.Monitor(monitor_tasks, session, global_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the optimisation\n",
    "We finally get to running the optimisation.\n",
    "\n",
    "We may want to continue a previously run optimisation by resotring the TensorFlow graph from the latest checkpoint. Otherwise skip this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "if os.path.isdir('./monitor-saves'):\n",
    "    mon.restore_session(session, './monitor-saves')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 10\ttotal itr.rate 12.02/s\trecent itr.rate 12.02/s\topt.step 10\ttotal opt.rate 13.49/s\trecent opt.rate 13.49/s\n",
      "Iteration 20\ttotal itr.rate 17.87/s\trecent itr.rate 34.80/s\topt.step 20\ttotal opt.rate 26.54/s\trecent opt.rate 820.59/s\n",
      "Iteration 30\ttotal itr.rate 22.46/s\trecent itr.rate 46.27/s\topt.step 30\ttotal opt.rate 38.90/s\trecent opt.rate 561.37/s\n",
      "Iteration 40\ttotal itr.rate 25.89/s\trecent itr.rate 47.68/s\topt.step 40\ttotal opt.rate 50.75/s\trecent opt.rate 592.97/s\n",
      "Iteration 50\ttotal itr.rate 28.44/s\trecent itr.rate 46.94/s\topt.step 50\ttotal opt.rate 62.08/s\trecent opt.rate 580.30/s\n",
      "Iteration 60\ttotal itr.rate 30.37/s\trecent itr.rate 45.99/s\topt.step 60\ttotal opt.rate 72.80/s\trecent opt.rate 530.95/s\n",
      "Iteration 70\ttotal itr.rate 31.95/s\trecent itr.rate 46.39/s\topt.step 70\ttotal opt.rate 83.31/s\trecent opt.rate 625.60/s\n",
      "Iteration 80\ttotal itr.rate 33.26/s\trecent itr.rate 46.73/s\topt.step 80\ttotal opt.rate 93.33/s\trecent opt.rate 588.08/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/100 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 90\ttotal itr.rate 34.27/s\trecent itr.rate 45.25/s\topt.step 90\ttotal opt.rate 102.96/s\trecent opt.rate 591.78/s\n",
      "Iteration 100\ttotal itr.rate 35.43/s\trecent itr.rate 50.97/s\topt.step 100\ttotal opt.rate 113.15/s\trecent opt.rate 1032.42/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 398.93it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 110\ttotal itr.rate 30.03/s\trecent itr.rate 11.91/s\topt.step 110\ttotal opt.rate 122.28/s\trecent opt.rate 632.37/s\n",
      "Iteration 120\ttotal itr.rate 30.81/s\trecent itr.rate 43.03/s\topt.step 120\ttotal opt.rate 129.67/s\trecent opt.rate 387.68/s\n",
      "Iteration 130\ttotal itr.rate 31.56/s\trecent itr.rate 44.48/s\topt.step 130\ttotal opt.rate 138.16/s\trecent opt.rate 643.12/s\n",
      "Iteration 140\ttotal itr.rate 32.36/s\trecent itr.rate 48.50/s\topt.step 140\ttotal opt.rate 147.09/s\trecent opt.rate 920.47/s\n",
      "Iteration 150\ttotal itr.rate 33.22/s\trecent itr.rate 52.80/s\topt.step 150\ttotal opt.rate 156.02/s\trecent opt.rate 1041.22/s\n",
      "Iteration 160\ttotal itr.rate 34.03/s\trecent itr.rate 53.58/s\topt.step 160\ttotal opt.rate 164.78/s\trecent opt.rate 1044.40/s\n",
      "Iteration 170\ttotal itr.rate 34.60/s\trecent itr.rate 47.35/s\topt.step 170\ttotal opt.rate 172.78/s\trecent opt.rate 772.81/s\n",
      "Iteration 180\ttotal itr.rate 35.05/s\trecent itr.rate 45.10/s\topt.step 180\ttotal opt.rate 179.82/s\trecent opt.rate 585.96/s\n",
      "Iteration 190\ttotal itr.rate 35.58/s\trecent itr.rate 48.83/s\topt.step 190\ttotal opt.rate 186.93/s\trecent opt.rate 648.86/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|████      | 40/100 [00:00<00:00, 392.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 200\ttotal itr.rate 35.98/s\trecent itr.rate 45.64/s\topt.step 200\ttotal opt.rate 194.89/s\trecent opt.rate 1019.55/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 412.92it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 210\ttotal itr.rate 33.91/s\trecent itr.rate 15.77/s\topt.step 210\ttotal opt.rate 201.35/s\trecent opt.rate 596.64/s\n",
      "Iteration 220\ttotal itr.rate 34.30/s\trecent itr.rate 45.05/s\topt.step 220\ttotal opt.rate 207.74/s\trecent opt.rate 622.73/s\n",
      "Iteration 230\ttotal itr.rate 34.59/s\trecent itr.rate 42.48/s\topt.step 230\ttotal opt.rate 212.04/s\trecent opt.rate 389.84/s\n",
      "Iteration 240\ttotal itr.rate 35.06/s\trecent itr.rate 51.18/s\topt.step 240\ttotal opt.rate 218.12/s\trecent opt.rate 640.71/s\n",
      "Iteration 250\ttotal itr.rate 35.33/s\trecent itr.rate 43.37/s\topt.step 250\ttotal opt.rate 222.00/s\trecent opt.rate 387.30/s\n",
      "Iteration 260\ttotal itr.rate 35.60/s\trecent itr.rate 43.86/s\topt.step 260\ttotal opt.rate 225.89/s\trecent opt.rate 401.48/s\n",
      "Iteration 270\ttotal itr.rate 35.95/s\trecent itr.rate 48.54/s\topt.step 270\ttotal opt.rate 232.38/s\trecent opt.rate 920.79/s\n",
      "Iteration 280\ttotal itr.rate 36.24/s\trecent itr.rate 46.05/s\topt.step 280\ttotal opt.rate 237.52/s\trecent opt.rate 589.08/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/100 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 290\ttotal itr.rate 36.50/s\trecent itr.rate 45.99/s\topt.step 290\ttotal opt.rate 241.64/s\trecent opt.rate 470.47/s\n",
      "Iteration 300\ttotal itr.rate 36.87/s\trecent itr.rate 52.17/s\topt.step 300\ttotal opt.rate 248.00/s\trecent opt.rate 1043.23/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 435.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 310\ttotal itr.rate 35.38/s\trecent itr.rate 16.01/s\topt.step 310\ttotal opt.rate 254.02/s\trecent opt.rate 938.13/s\n",
      "Iteration 320\ttotal itr.rate 35.70/s\trecent itr.rate 49.13/s\topt.step 320\ttotal opt.rate 257.69/s\trecent opt.rate 466.20/s\n",
      "Iteration 330\ttotal itr.rate 35.93/s\trecent itr.rate 45.17/s\topt.step 330\ttotal opt.rate 262.70/s\trecent opt.rate 696.29/s\n",
      "Iteration 340\ttotal itr.rate 36.18/s\trecent itr.rate 47.40/s\topt.step 340\ttotal opt.rate 267.10/s\trecent opt.rate 595.96/s\n",
      "Iteration 350\ttotal itr.rate 36.34/s\trecent itr.rate 42.88/s\topt.step 350\ttotal opt.rate 269.13/s\trecent opt.rate 362.87/s\n",
      "Iteration 360\ttotal itr.rate 36.68/s\trecent itr.rate 53.97/s\topt.step 360\ttotal opt.rate 274.80/s\trecent opt.rate 1046.37/s\n",
      "Iteration 370\ttotal itr.rate 36.97/s\trecent itr.rate 51.59/s\topt.step 370\ttotal opt.rate 278.99/s\trecent opt.rate 618.95/s\n",
      "Iteration 380\ttotal itr.rate 37.12/s\trecent itr.rate 43.85/s\topt.step 380\ttotal opt.rate 283.17/s\trecent opt.rate 635.62/s\n",
      "Iteration 390\ttotal itr.rate 37.25/s\trecent itr.rate 43.20/s\topt.step 390\ttotal opt.rate 284.76/s\trecent opt.rate 361.88/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 31%|███       | 31/100 [00:00<00:00, 308.62it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 400\ttotal itr.rate 37.46/s\trecent itr.rate 48.07/s\topt.step 400\ttotal opt.rate 288.56/s\trecent opt.rate 601.72/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 390.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 410\ttotal itr.rate 36.13/s\trecent itr.rate 14.87/s\topt.step 410\ttotal opt.rate 290.23/s\trecent opt.rate 377.88/s\n",
      "Iteration 420\ttotal itr.rate 36.30/s\trecent itr.rate 45.32/s\topt.step 420\ttotal opt.rate 294.90/s\trecent opt.rate 865.01/s\n",
      "Iteration 430\ttotal itr.rate 36.53/s\trecent itr.rate 49.97/s\topt.step 430\ttotal opt.rate 298.60/s\trecent opt.rate 631.63/s\n",
      "Iteration 440\ttotal itr.rate 36.78/s\trecent itr.rate 51.44/s\topt.step 440\ttotal opt.rate 302.12/s\trecent opt.rate 613.32/s\n",
      "Iteration 450\ttotal itr.rate 36.93/s\trecent itr.rate 44.99/s\topt.step 450\ttotal opt.rate 305.26/s\trecent opt.rate 562.42/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 46%|████▌     | 46/100 [00:00<00:00, 459.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 450\ttotal itr.rate 36.00/s\trecent itr.rate 0.00/s\topt.step 450\ttotal opt.rate 268.14/s\trecent opt.rate 0.00/s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 466.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tasks execution time summary:\n",
      "print:\t0.0385 (sec)\n",
      "model_tboard:\t0.2123 (sec)\n",
      "lml_tboard:\t1.2115 (sec)\n",
      "custom_tboard:\t1.2399 (sec)\n",
      "saver:\t4.0557 (sec)\n",
      "sleep:\t4.5382 (sec)\n"
     ]
    }
   ],
   "source": [
    "optimiser = gpflow.train.AdamOptimizer(0.01)\n",
    "\n",
    "with mon.Monitor(monitor_tasks, session, global_step, print_summary=True) as monitor:\n",
    "    optimiser.minimize(m, step_callback=monitor, maxiter=450, global_step=global_step)\n",
    "\n",
    "file_writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets compute the log likelihood again. Hopefully we will see an increase in its value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LML after the optimisation: -68705.124191\n"
     ]
    }
   ],
   "source": [
    "print('LML after the optimisation: %f' % m.compute_log_likelihood())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we have used the TensorFlow `AdamOptimizer`. Using `ScipyOptimizer` requires a couple of special tricks. Firstly, this optimiser works with its own copy of trained variables and updates the original ones only when the optimisation is completed. Secondly, it doesn't use the `global_step` variable. This can present a problem when doing optimisation in several stages. Monitor has to use an iteration count instead of the `global_step`, which will be reset to zero at each stage.\n",
    "\n",
    "To adress the first problem we will provide the optimiser as one of the parameters to the monitor. The monitor will make sure the orginal variables are updated whenever we access them from a monitoring task. The second problem is addressed by creating an instance of `MonitorContext` and providing it explicitely to the `Monitor`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Optimization terminated with:\n",
      "  Message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'\n",
      "  Objective function value: 15576.597541\n",
      "  Number of iterations: 5\n",
      "  Number of functions evaluations: 12\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Optimization terminated with:\n",
      "  Message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'\n",
      "  Objective function value: 15576.597541\n",
      "  Number of iterations: 5\n",
      "  Number of functions evaluations: 12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 5\ttotal itr.rate 5.46/s\trecent itr.rate nan/s\topt.step 5\ttotal opt.rate 5.46/s\trecent opt.rate nan/s\n",
      "Tasks execution time summary:\n",
      "print:\t0.0386 (sec)\n"
     ]
    }
   ],
   "source": [
    "optimiser = gpflow.train.ScipyOptimizer()\n",
    "context = mon.MonitorContext()\n",
    "\n",
    "with mon.Monitor([print_task], session, print_summary=True, optimiser=optimiser, context=context) as monitor:\n",
    "    optimiser.minimize(m, step_callback=monitor, maxiter=250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
